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This study evaluated the efficacy of the Word Learning Strategies (WLS) supplementary program 
designed to develop upper-elementary students’ vocabulary skills in order to improve reading 
comprehension. The study used a true, group-randomized, experimental design, which randomly 
assigned 46 4th grade classrooms (n=1324 students) from 12 districts to a treatment or control 
group. The results from the first cohort indicate that the program was positively associated with 
gains in students’ vocabulary learning as measured by Word Learning Strategies Test and in 
students’ reading comprehension as measured by Gates-MacGinitie Reading Test, after accounting 
for differences in baseline test scores. The use of the WLS program also led to increases in 
teachers’ awareness of strategies to support their students’ vocabulary and reading 
comprehension. 
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Study Overview 

A significant number of U.S. students do not develop the level of reading proficiency that 
they need to achieve in school, successfully join the increasingly knowledge-oriented workforce, 
and assist the U.S. in competing in the global economy. Reading is a complex process involving 
multiple interrelated components, and research conducted over the past 100 years has repeatedly 
shown that vocabulary is one of the most important of these components (Baumann, Kame'enui, & 
Ash, 2003; Beck & McKeown, 1991; Davis, 1944; Graves & Silverman, 2010; Thorndike, 1917). 
Vocabulary is also a central focus of major educational reform efforts such as Reading First (No 
Child Left Behind Act of 2001, 2002) and the Common Core State Standards (National Governors 
Association Center for Best Practices & Council of Chief State School Officers, 2010). 

Building a strong vocabulary requires learning a large number of words. Based on the work 
of Anglin (1993), Snow and Kim (2007), and Stahl and Nagy (2006), our estimate is that average 
twelfth graders know approximately 50,000 words and that students therefore learn close to 3,000 
to 4,000 words each year. Clearly, this is far more words than can be directly taught, and students 
need to develop strategies for learning words on their own. 

These estimates are for typical students whose first language is English. As is widely 
recognized, attaining strong vocabularies is particularly challenging for many English learners 
(August, Carlo, Dressler, & Snow 2005) and a number of children from low income families 
(Becker, 1977; Chall, Jacobs, & Baldwin, 1990; Hart & Risley, 1995). Therefore, while learning to 
effectively and efficiently use word learning strategies is crucial for all students, learning such 
strategies is particularly crucial for English learners and for children from low income 
backgrounds. 

This study focuses on the Word Learning Strategies (WLS) supplementary curriculum that 
is designed to develop upper-elementary students’ vocabulary acquisition skills in order to improve 
reading comprehension. The study addresses how the WLS program is implemented in elementary 


schools with high numbers of English learners (ELs) and students from low-income backgrounds. 
It tests the educational efficacy of the WLS program in increasing 4th grade students’ vocabulary 
learning and reading comprehension. In addition, the study discusses the implications for 
vocabulary instructional practice. 


Description of Word Learning Strategies Program 

WLS is a supplemental program for teaching word learning strategies. The program was 
developed under a Small Business Innovation Research (SBIR) development study awarded by the 
Institute of Education Sciences (IES), and all materials are fully developed and available for 
conducting an efficacy study. The program includes a set of practical, research-based, and 
theoretically sound strategies for inferring the meanings of unknown words that students encounter 
while reading, thereby increasing their ability to derive meaning while reading independently 
(Duffy et al., 1986; Duke & Pearson, 2002; Pressley, Harris, & Marks, 1992; Wharton-McDonald, 
2006; Duke, Pearson, Strachan, & Billman, 2011). Students are taught to use context clues, word 
parts, and the dictionary to learn the meaning of unknown words. Spanish-speaking ELs receive 
additional instruction in using cognates, and all ELs receive instruction in recognizing idioms. 

The basic model for instruction is teacher-led direct explanation with constructivist 
elements, an approach explained below. The program is intended for all students—ELs, average 
learners, above average learners, and less proficient learners. The program prepares teachers to 
teach word learning strategies and to explain to students why the strategies are important for 
reading. Teacher materials include: (1) online tutorials, including videos, to prepare teachers to use 
the materials; (2) a detailed teacher manual with day-by-day lesson plans, a teacher reflection log, 
and instructions for using the online system for supplemental lessons; and (3) presentation 
materials (e.g., slides for overhead projecting, posters, game cards). 

Student materials include activity books, quizzes, and tests. In addition to strategy practice 
with individual words and sentences, larger passages of authentic text are provided so that students 
can practice using these strategies as they would while reading independently. To provide extra and 
differentiated assistance for students who need it, the program includes supplementary, web-based 
instruction and games for each of the strategies, instruction on using cognates (for Spanish- 
speaking ELs), and instruction on recognizing and understanding idioms (for all ELs). 

The program provides 15 weeks of whole-class instruction for a typical 4th or Sth grade 
class, an additional 22 remedial, web-based lessons for students who need more practice, three 
web-based lessons on Spanish cognates for Spanish-speaking EL students, and three web-based 
lessons on idioms for all ELs. The whole-class instruction is delivered three days a week for about 
30 minutes per day throughout the 15-week period. 

The teacher manual includes four main instructional sections. The Word Parts Unit (seven 
weeks) provides lessons for teaching students how to identify and use morphology (inflectional 
suffixes, prefixes, derivational suffixes, roots, and compound words) to derive the meaning of 
unknown words they encounter as they read independently. The Context Unit (five weeks) 
provides lessons for teaching students to infer the meaning of unknown words from linguistic 
context clues (definition, synonym, antonym, and general clues). The Dictionary Unit (one week) 
provides lessons for teaching students to effectively use dictionaries as they are reading to identify 
the meaning of unknown words. The Combined Strategy Unit (two weeks) provides lessons for 
teaching students to combine word parts, context, and dictionary strategies to derive the meaning of 
unknown words. In each lesson plan, the teacher’s guide provides key elements of successful 
instruction: 


1. Key Messages: The points to be emphasized with students during the lesson (e.g., “You 
can use smaller words inside compound words to explain their meanings.’’); 

2. Objectives: A description of what students will be able to do by the end of the lesson 
(e.g., “Define compound word.”’); 

3. Lesson at a Glance: A quick overview of the predictable and consistent lesson structure 
(A. Focus, B. Teach, C. Practice/Apply, D. Wrap Up) with the number of minutes 
needed for each part of the lesson; and 

4. Materials and Equipment: A list of supplies needed for the lesson. 


Each lesson in the guide begins with a brief “Focus Activity” designed to capture students’ 
attention and motivate them to learn. This may be in the form of a quick game, some thought- 
provoking questions, or a brief review. The main instructional activities, which are the bulk of the 
lesson, are the “Teach and Practice/Apply” activities. Time devoted to these activities varies 
depending on where students are in each unit. In the earlier unit lessons, teachers devote more time 
to teaching, modeling, and guiding. As the unit progresses, direct teaching time decreases, and the 
time dedicated to practice and application increases. Assessment occurs every two to three weeks. 
The final part of each lesson is the “Wrap Up” section, during which teachers bring the lesson to a 
close, provide corrective feedback, summarize what students learned, and/or give students a chance 
to reflect on their learning. A speech balloon icon in the teacher’s guide signals the sample teaching 
language that is provided to offer suggestions for explaining strategies, giving directions, posing 
questions, and interacting with students. In addition, a computer monitor icon marks the activities 
that have accompanying video in the web-based teacher training. 

As noted, the pedagogy used in the whole-class instruction is a combination of two widely 
researched and recommended approaches. The first approach—direct explanation of strategies— 
includes: (1) an explicit description of the strategy and when and how it should be used; (2) teacher 
and/or student modeling of the strategy in action; (3) collaborative use of the strategy in action; (4) 
guided practice using the strategy with gradual release of responsibility; and (5) independent use of 
the strategy (Duffy et al., 1986; Duke & Pearson, 2002; Duke, Pearson, Strachan, & Billman, 
2011). The second approach—the use of constructivist elements—is primarily motivated by the 
work of Pressley and his colleagues (Pressley, Harris, & Marks, 1992; Wharton-McDonald, 2006), 
who found that successful use of direct explanation typically involves constructivist elements. 
Those in WLS include: (1) motivating students to use the strategies; (2) discussing with students 
the value of the strategies; (3) providing verbal explanations and collaborative discussion of the 
thinking processes associated with strategy steps; (4) providing extensive feedback and engaging in 
substantial collaborative discussion as students try strategies; and (5) extending instruction and 
practice over a long period of time and across diverse tasks. The inclusion of these constructivist 
elements is further prompted by the importance of motivation as recognized by the National 
Research Council (1999) and reading theorists such as Guthrie, Wigfield, and Perencevich (2004), 
as well as by modern theories of transfer such as those of Engle (2012) and Perkins and Salomon 
(2012). 


Logic Model 

The study’s logic model (Figure 1) posits that implementation of the 15 weeks of WLS 
curriculum, along with its web-based interactive games, will improve students’ use of WLS in 
reading passages, their vocabulary, their reading comprehension, and eventually their school 
achievement. Specifically, the study addressed the following research questions: 


1. Does the WLS intervention increase vocabulary knowledge for 4" graders? 


2. Does the WLS intervention improve reading proficiency for 4" graders? 
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Figure 1. WLS Logic Model 


Study Design and Methodology 

Study Method 

This study was implemented over two years, with a separate cohort of classrooms participating 
each year. The study used a true, group-randomized, experimental design, which randomly 
assigned 4" grade classrooms to a treatment or control group. Classrooms randomly assigned to be 
in the treatment group implemented the WLS supplementary curriculum; whereas classrooms 
randomly assigned to be in the control condition implemented their usual English Language Arts 
practices. 

To prepare treatment teachers to implement the WLS program effectively and with fidelity, the 
research team facilitated an interactive, online training for the teachers. The online training focused 
on: (1) background information about vocabulary development, instruction, and the use of WLS; (2) 
a demonstration of useful classroom practices; and (3) key components of curriculum 
implementation (e.g., following the teacher manual, dosage, pacing). 

Student outcome data on two reading assessments were collected at baseline and endline for 
treatment and control groups. Teachers were trained in how to administer these outcome measures 
effectively. In addition to collecting student outcome data, teacher baseline and endline data were 
collected for both treatment and control groups on a survey of teacher content knowledge of 


teaching reading. Throughout the study, fidelity of implementation was monitored using weekly 
activity logs,classroom observations, and teacher interviews. 

To analyze the impact of the program, two-level hierarchical linear modeling (HLM) was 
applied to analyze student outcomes. Qualitative data collected from open-ended survey questions, 
teacher logs, and interviews were analyzed using grounded theory, or constant comparative 
analysis (Strauss and Corbin, 1998). In an initial data reduction approach, respondents’ comments 
were reviewed and assigned categories of meaning (open coding). Then, these categories along 
with quantitative data results were reviewed for causal linkages and non-causal relationships 
related to the central phenomenon (axial coding), which allowed the researcher to develop a “story’ 
that connects the categories (selective coding) and, finally, posit hypotheses or theoretical 
propositions. These qualitative analyses provided descriptions of: 1) how teachers implemented the 
WLS program, 2) how the WLS curriculum, along with its digital lessons, may increase teachers’ 
instructional practice and support of their students’ vocabulary learning and reading 
comprehension. 


> 


Participants 

This paper focuses on cohort | of the WLS study, which participated during the 2016-17 
school year. A total of 50 4th grade classrooms from 12 diverse districts throughout the state of 
CA were randomized to a treatment (n=25) or control (n=25) condition. Pre and post student 
assessment data was collected on 1315 students from 45 classrooms in 12 districts. Students who 
were not 4th graders, opted out of the study, transferred out of the school, or moved into 
participating classrooms after the intervention began were excluded from the final analytic 
sample. The analytic sample includes 1083 students from 23 treatment classrooms and 21 control 
classrooms in 11 districts. 

Instruments 

Measures of implementation fidelity 

In order to monitor and measure fidelity of implementation, we collected weekly teacher 
logs of classroom activity, as well as conducted classroom observations and interviews. 

(1) Teacher Logs: These logs were designed to measure the extent to which participating 
teachers covered WLS concepts and used WLS instructional strategies. The logs were aligned 
with the WLS curriculum to provide a measure of fidelity of implementation. General reporting 
categories on the teacher log included: (a) amount of teaching time devoted to WLS; (b) use of 
various WLS teaching strategies; (c) teachers’ perceptions related to student understanding; and 
(d) questions related to any problems or issues that teachers encountered during implementation, 
including questions on pacing of the lessons and use of supplementary materials for EL students. 

(2) Classroom Observations and Teacher Interviews: Classroom observations were 
conducted in 20 classrooms, including 15 treatment and 5 control teachers, representing a range 
of schools, districts, and geographic areas. The classroom observations were designed to allow 
documentation of: (a) the WLS components covered in the lesson; (b) resources and equipment 
used; (c) classroom setup; and (d) a snapshot of student activities. Teacher interviews focused 
on: (a) teachers’ use of the WLS curriculum; (b) student engagement and learning; and (c) 
feedback on the WLS training. On the observation protocol for the treatment teachers, 
researchers noted the classroom setup; the WLS components taught in the lesson (e.g., 
Practice/Apply/Guide); equipment and other resources used for the WLS lesson; and a running 
record in which the researchers captured as much data as possible about the timing of the 
different lesson components, modes of instruction, student and teacher discourse, and level of 


student participation. For the control teachers, the observation protocol included a description of 
any vocabulary component in the lesson that was observed; the equipment and resources used 
for the lesson; modes of instruction; teacher discourse; student discourse; and level of student 
engagement. 


Student measures 

In order to get a broader picture of student achievement, we used separate quantitative 
measures of student knowledge: (1) The Word Learning Strategies Test (WLS Test) and (2) The 
Gates-MacGinitie Reading Test (GMRT). 

(1) The WLS Test is a 34-item test created by the developer of the intervention. It includes 
closed and open-ended items, and assesses student knowledge of prefixes, suffixes, context cues, 
as well as the Word Parts Strategy, the Dictionary Strategy, and the Combined Strategy. The 
measure also assesses students’ ability to apply the Word Parts, Context, and Dictionary 
Strategies to highlighted words presented in the context of short stories. Thirty-five percent of 
the assessment tests knowledge, and 65% tests application. Data collected during the SBIR pilot 
test of the WLS intervention indicate that the instrument has good reliability. Specifically, 
Cronbach’s alpha for the entire instrument ranged from 0.875 at pre-test to 0.921 at post-test. 

(2) The GMRT (MacGinitie, MacGinitie, Maria, & Dreyer, 2002) is a series of 
standardized, multiple choice, norm-referenced tests of reading achievement that can be 
delivered in a paper/pencil format or online. The GMRT for grades 3-12 includes two subtests— 
vocabulary and comprehension. The difficulty level of the questions on the GMRT progresses 
from easy in the beginning to difficult at the end. Each level of the GMRT is designed to 
accurately measure performance across a range of reading levels. Kuder-Richardson Formula 20 
(KR-20) was utilized to assess the reliability index for the subtests. Internal reliability 
coefficients were 0.80 for the vocabulary subtest and 0.90 for the comprehension subtest. 


Results 

Program Implementation 

Prior to implementing the WLS supplemental curriculum, the 23 treatment teachers 
participated in approximately 5 hours of professional development provided by WestEd. WestEd 
conducted a two-hour online webinar with teachers that provided background information about 
vocabulary development, instruction, and the use of WLS. The WestEd research team also 
described the research tasks associated with the study and how teachers were to collect student 
assessment data. Teachers then participated in an hour-long online module on their own time. 


Implementation fidelity 

In order to monitor teachers’ implementation of WLS—particularly as it related to dosage 
and pacing—treatment teachers completed a weekly teacher log. Collectively, the teachers 
implemented WLS an average of three to four days per week. The majority of the teachers were 
able to implement at least 90% of the curriculum. However, only 27% of cohort 1 teachers and 
12% of cohort 2 teachers were able to implement all curriculum lessons in 15 weeks (Figure 2). 
This is due to time demands, such as class schedule changes, district adoption of other curricula, 
natural disasters, district testing, winter plays, fire drills, school minimum days, and vacations. 

In order to monitor teachers’ implementation of WLS—particularly as it related to dosage 
and pacing—treatment teachers completed a weekly teacher log. The teacher logs suggested that 
teachers were implementing WLS for the appropriate duration and at the recommended pace. 


Collectively, the teachers implemented WLS on average three days per week, which is the 
recommended dosage. Teachers, in general, were able to follow the pacing guides and 
implemented all four units of WLS curriculum, although some teachers skipped some lessons 
because of their class schedules. Table 1 shows the average implementation dosage of cohort 1 
treatment teachers. 


Table 1. Cohort 1 Implementation Dosage 


Word Parts (21 lessons) 10-21 17 
Context (15 lessons) 0-15 12 
Dictionary (3 lessons) 0-3 2.4 
Combined (6 lessons) 0-6 5 
Overall (45 lessons) 17-45 36 


Integration of WLS into existing instructional practices 

Researchers conducted direct classroom observations of 15 treatment teachers and 5 control 
teachers. After each observation, a brief teacher interview was conducted to obtain feedback about the 
WLS curriculum. The observations and interviews suggested that the control teachers reported that their 
supplementary vocabulary programs tended to concentrate on teaching individual words. In contrast, the 
treatment teachers felt that the WLS curriculum was Common Core aligned; was appropriate for 4th 
grade students; was easy to learn and to implement; and was beneficial to all students including their 
ELs, as it provided focused, sustained, and in-depth instruction in word learning strategies. Most 
treatment teachers were able to learn the WLS curriculum and implement it correctly without excessive 
additional preparation or support. Treatment teachers reported that the WLS curriculum was easy to 
integrate into their lesson planning. Most treatment teachers were successfully able to teach all of the 
WLS lesson components, use the appropriate materials, display adequate subject-matter knowledge, and 
keep students engaged. The area most challenging was pacing, as teachers reported that some planned 
WLS lessons could not be implemented as scheduled because of other school demands. 


Students’ Vocabulary and Reading Comprehension 
To investigate the impact on WLS curriculum on student outcomes, the following multilevel model 
was fitted to each outcome: 


Outcome;; = Bo; + By;Treatment;; + BzStuPretest;; + B3,female;; + Balatino;; 
+ B,)FRL;; + BelEP,; + B7,SBAC Math;; + Bg SBAC ELA; + 1; 


Boj = Goo + GosSitePretestMean,; + Uo; 
By; = Gyo + Uy; 


Ugj™~ N(O, Too) 
uy;~ N(0,T31) 


where: 

Goo is the overall grand average outcome score 

Go; is the expected change in school average score when school pretest changes 
Go is the average treatment of WLS 


B2_,; are the average within-school relationships between student characteristics and thr outcome 
and 


Too is the variance capturing the extent to which the schoool posttest means vary 
T,, is the variance capturing the extent to which the treatment effects vary between schools 


The model parameters of interest are Gio, which describes the average treatment effect, and 
TT 11, which can help give an estimate of the possible range of treatment effects in the population of 
schools. With this model, we treat each school as if it were its own “mini-experiment” evaluating 
the impact of WLS on various outcomes to estimate an average WLS treatment effect. 

The results indicate that the intervention was positively associated with gains in students’ 
vocabulary learning as measured by WLS assessment (point estimate of 4.56). This difference was 
significant at the .01 level, after accounting for differences in baseline test results and student 
demographic characteristics (e.g., race/ethnicity, Free Reduced Lunch program qualification, 
gender, and Special Education program enrollment). However, the intervention was not 
significantly associated with gains in students’ reading comprehension as measured by GMRT (see 
Table 2). One possible explanation for these results is that the WLS intervention focused primarily 
on word parts, context clues, use of dictionary, and the combination of these strategies, which were 
measured more directly by the WLS assessment. Although GMRT includes both vocabulary and 
reading comprehension components, the test items did not directly assess students’ use of word 
learning strategies. Therefore, GMRT may not be sensitive enough to detect changes in the short- 
term. 

Table 2. Student Outcomes as Measured by WLS Assessment and GMRT 


Model 2: WLS Outcome Model 3: GMRT Outcome 

Est. SE. P-val Est. SE. P-val 
GOO: Grand Mean 6.80 137° (0.00 -*** 7.45 38.14 0.85 
G01: School Pretest 0.88 0.09 0.00 *** 1.01 0.08 0.00 *** 
G10: Treatment Effect 4.56 0.64 0.00 *** 0.81 2.70 0.77 
B2: Student Pretest 0.30 0.03 0.00 *** 0.64 0.03 0:00. *** 
B3: Female 0.06 0.29 0.83 0.48 1.08 0.66 
B4: Latino -1.12 0.36 0.00 ** -2.16 133. -0.11 


B5: FRL -0.26 0.35 0.46 1.16 1.28 0.37 


B6: IEP -0.93 0.55 0.09 . -0.50 2.22. 0.82 

B7: SBAC Math 0.03 0:00' 0:00 #** 0.04 0.01 0.00 *** 

B8: SBAC ELA 0.02 0.00 0.00 *** 0.10 | 0.01 0.00 *** 
Var SD ICC Variance sD ICC 

TOO 1.163 1.078 5% 14.18 3.765 4% 

Ti 5.442 2.333 23% 96.52 9.824 30% 

R 16.83 4.102 72% 215.79 14.69 66% 

N Students 825 773 

N Schools 17 17 


p < 0.001 ***; p< 0.01 **; p< 0.05 *;p<0.10. 


Implications and Future Study 

Importance of Communication with Teachers 

Throughout the study, the WestEd research team maintained frequent communication with all 
23 teachers implementing the WLS curriculum. To facilitate communication with the teachers, the 
research team created a shared email account which was consistently monitored to ensure prompt 
replies to teacher inquiries. Tailored weekly newsletters were sent out through the shared email 
account to inform teachers of study updates, including additional resources, helpful tips, and 
stipend delivery. Timely responses to questions or concerns were critical to prevent teacher 
confusion about implementation of the curriculum, which could have affected implementation 
fidelity or resulted in the collection of unreliable data. 

The weekly teacher logs allowed researchers to examine whether teachers were implementing 
the curriculum as intended, enabling researchers to offer support to teachers who were 
experiencing issues with implementation. In addition to using teacher logs to collect information 
about the number of lessons completed or the time spent per lesson, the logs also provided insight 
about the level of teacher engagement. WestEd researchers aimed to maintain a high level of 
teacher engagement, to minimize participant fatigue, and to prevent study attrition. Moving into 
cohort 2 of the study, researchers will continue to emphasize the importance of the teachers’ role 
throughout the study and acknowledge their efforts in contributing to a body of research around 
elementary vocabulary instruction. 


Looking Forward: Future Analysis 

The results of the cohort 1 study indicate that by focusing on one facet of vocabulary instruction— 
teaching word learning strategies—the WLS curriculum responds to the need for teachers to be 
equipped with powerful yet straightforward ways to help their students improve their vocabularies 
which in turn has the potential to improve their reading comprehension, their motivation, and their 
overall success in school. The WLS curriculum provides clear guidance and appropriate materials 
so that teachers can successfully implement the curriculum and track the progress of their students 
as they do so. To further address the efficacy of the WLS curriculum, we are currently conducting 
cohort 2 of the study involving 48 teachers. As we move forward with the study, we will combine 
the data from the two cohorts to further understand the impact of WLS on students’ vocabulary and 
reading comprehension. Since cohort | results indicated that there was considerable variability of 


the treatment effect across the sites ICC=23% for WLS assessment and ICC=30% for GMRT 
test), we will further analyze the school context and conduct sub-group analyses to explore whether 
WLS has different impacts on students from different school contexts. 
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